Intelligent metamaterial radar for target identification

ABSTRACT

Examples disclosed herein relate to an Intelligent Metamaterial (“iMTM”) radar for target identification. The iMTM radar has an iMTM antenna module to radiate a transmission signal with an iMTM antenna structure and generate radar data capturing a surrounding environment. An iMTM interface module detects and identifies a target in the surrounding environment from the radar data and controls the iMTM antenna module.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to PCT Application No. PCT/US18/30541,filed on May 1, 2018, which itself claims priority to U.S. ProvisionalApplication No. 62/515,045, filed on Jun. 5, 2017. This application alsoclaims priority to U.S. Provisional Application No. 62/613,675, filed onJan. 4, 2018, U.S. Provisional Application No. 62/651,050, filed on Mar.30, 2018, U.S. Provisional Application No. 62/663,243, filed on Apr. 26,2018, and U.S. Provisional Application No. 62/666,666, filed on May 3,2018. These applications are incorporated herein by reference.

BACKGROUND

Autonomous driving is quickly moving from the realm of science fictionto becoming an achievable reality. Already in the market areAdvanced-Driver Assistance Systems (“ADAS”) that automate, adapt andenhance vehicles for safety and better driving. The next step will bevehicles that increasingly assume control of driving functions such assteering, accelerating, braking and monitoring the surroundingenvironment and driving conditions to respond to events, such aschanging lanes or speed when needed to avoid traffic, crossingpedestrians, animals, and so on.

An aspect of making this work is the ability to detect and classifytargets in the surrounding environment at the same or possibly evenbetter level as humans. Humans are adept at recognizing and perceivingthe world around them with an extremely complex human visual system thatessentially has two main functional parts: the eye and the brain. Inautonomous driving technologies, the eye may include a combination ofmultiple sensors, such as camera, radar, and lidar, while the brain mayinvolve multiple artificial intelligence, machine learning and deeplearning systems. The goal is to have full understanding of a dynamic,fast-moving environment in real time and human-like intelligence to actin response to changes in the environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The present application may be more fully appreciated in connection withthe following detailed description taken in conjunction with theaccompanying drawings, in which like reference characters refer to likeparts throughout, and in which:

FIG. 1 illustrates a schematic diagram of an iMTM radar system for usein an autonomous driving system in accordance with various examples;

FIG. 2 is a schematic diagram of an iMTM antenna structure for use withthe iMTM radar system of FIG. 1 in accordance with various examples;

FIG. 3 is a flowchart illustrating the operation of an example iMTMradar system in more detail;

FIG. 4 illustrates the encoding of 4D radar data into a point cloud inaccordance with various examples;

FIG. 5 illustrates an example data encoding to generate a point cloudfrom 4D radar data;

FIG. 6 illustrates other example data encodings to generate a pointcloud from 4D radar data;

FIG. 7 is a schematic diagram of a NLOS correction module for use in adata pre-processing module in an iMTM radar in accordance with variousexamples;

FIG. 8 is a flowchart illustrating the operation of the NLOS correctionmodule of FIG. 6;

FIG. 9 is a schematic diagram of an iMTM interface module of FIG. 1 inaccordance with various examples;

FIG. 10 is a flowchart illustrating the steps for training the CNN inthe target detection module of FIG. 9;

FIG. 11 is a flowchart for training the DNN of the iMTM interface moduleshown in FIG. 9;

FIG. 12 is a schematic diagram of an autonomous driving system having aniMTM radar in accordance with various examples;

FIGS. 13-14 illustrate processes implemented in the sensor fusion moduleof FIG. 12;

FIG. 15 is a schematic diagram of an example iMTM antenna structure inaccordance with various examples;

FIG. 16 illustrates an example iMTM antenna array for use in an iMTMantenna structure;

FIGS. 17-18 are schematic diagrams of example iMTM antenna structures;

FIG. 19 is a schematic diagram of an example iMTM antenna array; and

FIG. 20 is another perspective of the iMTM antenna array of FIG. 19illustrating its various layers in accordance with various examples.

DETAILED DESCRIPTION

Systems and methods for an Intelligent Metamaterial (“iMTM”) radar fortarget detection and identification are disclosed. The systems andmethods are suitable for many different applications and can be deployedin a variety of different environments and configurations. In variousexamples, the systems and method are used in an autonomous drivingvehicle to detect and identify targets in the vehicle's path andsurrounding environment. The targets may include structural elements inthe environment such as roads, walls, buildings, road center medians andother objects, as well as vehicles, pedestrians, bystanders, cyclists,plants, trees, animals and so on.

The iMTM radar is a “digital eye” with true 3D vision and capable of ahuman-like interpretation of the world. The digital eye and human-likeinterpretation capabilities are provided by two main modules: an iMTMAntenna Module and an iMTM Interface Module. The iMTM antenna module isbased on a dynamically controllable antenna structure with metamaterialantenna arrays capable of providing a 360° view of a vehicle's path andsurrounding environment. The iMTM interface module receives data fromthe iMTM antenna module corresponding to a Field of View (“FoV”) and istrained to detect and identify targets thereon. The iMTM interfacemodule can also control the iMTM antenna module as desired.

In various examples, the control of the iMTM antenna module may involvechanging the electrical or electromagnetic configuration of themetamaterial antenna arrays. This may be accomplished, for example, withthe use of varactors to enable adjustment of radiation patterns from theantenna arrays in the iMTM antenna module. Each antenna array is anarray of individual antenna elements including intelligent metamaterialcells (“iMTM cells”). In various examples, the iMTM cells may beconfigured into subarrays that have specific characteristics.

For use in an autonomous driving vehicle, the iMTM radar system mayperform a coarse focus with a large beam width as an ambient condition,and then narrow the beam width when an echo is received, indicating atarget is within the FoV of the radiation patterns. In this way, thelarger beam width may sweep the full FoV of the iMTM antenna module,reducing the time to scan the FoV. In some examples, the iMTM interfacemodule is able to detect the area of the FoV showing a target and mapthat to a specific configuration of iMTM cells and/or subarrays to focusa beam, i.e., narrow the beam width. Additionally, in some examples, thespecific dimensions and other properties of a detected target, such astraveling velocity with respect to the antenna structure, are analyzedand a next action(s) or course of action(s) is determined. The detectedtarget in some examples is then provided as a visual or graphic display,which may act as a back-up security feature for a passenger in theautonomous driving vehicle.

It is appreciated that, in the following description, numerous specificdetails are set forth to provide a thorough understanding of theexamples. However, it is appreciated that the examples may be practicedwithout limitation to these specific details. In other instances,well-known methods and structures may not be described in detail toavoid unnecessarily obscuring the description of the examples. Also, theexamples may be used in combination with each other.

FIG. 1 illustrates a schematic diagram of an iMTM radar system for usein an autonomous driving system in accordance with various examples.iMTM radar system 100 is a “digital eye” with true 3D vision and capableof a human-like interpretation of the world. The “digital eye” andhuman-like interpretation capabilities are provided by two main modules:iMTM Antenna Module 102 and iMTM Interface Module 104.

iMTM antenna module 102 has an iMTM antenna structure 106 to radiatedynamically controllable and highly-directive RF beams usingmetamaterials. A transceiver module 108 coupled to the iMTM antennastructure 106 prepares a signal for transmission, such as a signal for aradar device, wherein the signal is defined by modulation and frequency.The signal is provided to the iMTM antenna structure 106 through acoaxial cable or other connector and propagates through the structurefor transmission through the air via RF beams at a given phase,direction, and so on. The RF beams and their parameters (e.g., beamwidth, phase, azimuth and elevation angles, etc.) are controlled byantenna controller 110, such as at the direction of iMTM interfacemodule 104.

The RF beams reflect off of targets in the vehicle's path andsurrounding environment and the RF reflections are received by thetransceiver module 108. Radar data from the received RF beams isprovided to the iMTM interface module 104 for target detection andidentification. A data pre-processing module 112 processes the radardata to encode it into a point cloud for the iMTM interface module 104.In various examples, the data pre-processing module 112 could be a partof the iMTM antenna module 102 or the iMTM interface module 104, such ason the same circuit board as the other modules within the iMTM antennaor interface modules 102-104. The radar data may be organized in sets ofRange-Doppler (“RD”) map information, corresponding to 4D informationthat is determined by each RF beam radiated off of targets, such asazimuthal angles, elevation angles, range and velocity. The RD maps maybe extracted from Frequency-Modulated Continuous Wave (“FMCW”) radarpulses and contain both noise and systematic artifacts from Fourieranalysis of the pulses. The iMTM interface module 104 controls furtheroperation of the iMTM antenna module 102 by, for example, providing beamparameters for the next RF beams to be radiated from the iMTM cells inthe iMTM antenna structure 106.

In operation, the antenna controller 110 is responsible for directingthe iMTM antenna structure 106 to generate RF beams with determinedparameters such as beam width, transmit angle, and so on. The antennacontroller 110 may, for example, determine the parameters at thedirection of iMTM interface module 104, which may at any given time wantto focus on a specific area of a FoV upon identifying targets ofinterest in the vehicle's path or surrounding environment. The antennacontroller 110 determines the direction, power, and other parameters ofthe beams and controls the iMTM array structure 106 to achieve beamsteering in various directions. The antenna controller 110 alsodetermines a voltage matrix to apply to capacitance control mechanismscoupled to the iMTM array structure 106 to achieve a given phase shift.In some examples, the iMTM array structure 106 is adapted to transmit adirectional beam through active control of the reactance parameters ofthe individual iMTM cells that make up the iMTM antenna structure 106.iMTM interface module 104 provides control actions to the antennacontroller 110 at the direction of the Target Identification andDecision Module 114.

Next, the iMTM antenna structure 106 radiates RF beams having thedetermined parameters. The RF beams are reflected off of targets in andaround the vehicle's path (e.g., in a 360° field of view) and arereceived by the transceiver module 108 in iMTM antenna module 102. Theantenna module 102 then transmits 4D radar data to the datapre-processing module 112 for generating a point cloud that is then sentto the iMTM interface module 104. A micro-doppler module 116 coupled tothe iMTM antenna module 102 and the iMTM interface module 104 extractsmicro-doppler signals from the 4D radar data to aid in theidentification of targets by the iMTM interface module 104. Themicro-doppler module 116 takes a series of RD maps from the iMTM antennamodule 102 and extracts a micro-doppler signal from them. Themicro-doppler signal enables a more accurate identification of targetsas it provides information on the occupancy of a target in variousdirections. Non-rigid targets such as pedestrians, cyclists are known toexhibit a time-varying doppler signature due to swinging arms, legs,etc. By analyzing the frequency of the returned radar signal over time,it is possible to determine the class of the target (i.e., whether avehicle, pedestrian, cyclist, animal, etc.) with over 90% accuracy.Further, as this classification may be performed by a linear SupportVector Machine (“SVM”), it is extremely computationally efficient. Invarious examples, the micro-doppler module 116 could be a part of theiMTM antenna module 102 or the iMTM interface module 104, such as on thesame circuit board as the other modules within the iMTM antenna orinterface modules 102-04.

The target identification and decision module 114 receives the pointcloud from the data pre-processing module 112, processes the point cloudto detect and identify targets, and determines the control actions to beperformed by the iMTM antenna module 102 based on the detection andidentification of such targets. For example, the target identificationand decision module 114 may detect a cyclist on the path of the vehicleand direct the iMTM antenna module 102, at the instruction of itsantenna controller 110, to focus additional RF beams at a given phaseshift and direction within the portion of the FoV corresponding to thecyclist's location.

The iMTM interface module 104 also includes a multi-object tracker 118to track the identified targets over time, such as, for example, withthe use of a Kalman filter. The multi-object tracker 118 matchescandidate targets identified by the target identification and decisionmodule 114 with targets it has detected in previous time windows. Bycombining information from previous measurements, expected measurementuncertainties, and some physical knowledge, the multi-object tracker 118generates robust, accurate estimates of target locations.

Information on identified targets over time are then stored at a TargetList and Occupancy Map 120, which keeps tracks of targets' locations andtheir movement over time as determined by the multi-object tracker 118.The tracking information provided by the multi-object tracker 118 andthe micro-doppler signal provided by the micro-doppler module 116 arecombined to produce an output containing the type of target identified,their location, their velocity, and so on. This information from iMTMradar system 100 is then sent to a sensor fusion module (described inmore detail below with reference to FIG. 12) in the vehicle, where it isprocessed together with information from other sensors in the vehicle.

In various examples, an FoV composite data unit 122 stores informationthat describes a FoV. This may be historical data used to track trendsand anticipate behaviors and traffic conditions or may be instantaneousor real-time data that describes the FoV at a moment in time or over awindow in time. The ability to store this data enables the iMTMinterface module 104 to make decisions that are strategically targetedat a particular point or area within the FoV. For example, the FoV maybe clear (no echoes received) for five minutes, and then one echoarrives from a specific region in the FoV; this is similar to detectingthe front of a car. In response, the iMTM interface module 104 maydetermine to narrow the beam width for a more focused view of thatsector or area in the FoV. The next scan may indicate the targets'length or other dimension, and if the target is a car, the iMTMinterface module 104 may consider what direction the target is movingand focus the beams on that area. Similarly, the echo may be from aspurious target, such as a bird, which is small and moving quickly outof the path of the car. There are a variety of other uses for the FoVcomposite data 122, including the ability to identify a specific type oftarget based on previous detection. A memory 124 stores useful data foriMTM radar system 100, such as, for example, information on whichsubarrays of the iMTM antenna structure 106 perform better underdifferent conditions.

In various examples described herein, the use of iMTM radar system 100in an autonomous driving vehicle provides a reliable way to detecttargets in difficult weather conditions. For example, historically adriver will slow down dramatically in thick fog, as the driving speeddecreases with decreases in visibility. On a highway in Europe, forexample, where the speed limit is 115 km/h, a driver may need to slowdown to 40 km/h when visibility is poor. Using the iMTM radar system100, the driver (or driverless vehicle) may maintain the maximum safespeed without regard to the weather conditions. Even if other driversslow down, a vehicle enabled with the iMTM radar system 100 will be ableto detect those slow-moving vehicles and obstacles in the way andavoid/navigate around them.

Additionally, in highly congested areas, it is necessary for anautonomous vehicle to detect targets in sufficient time to react andtake action. The examples provided herein for an iMTM radar systemincrease the sweep time of a radar signal so as to detect any echoes intime to react. In rural areas and other areas with few obstacles duringtravel, the iMTM interface module 104 adjusts the focus of the beam to alarger beam width, thereby enabling a faster scan of areas where thereare few echoes. The iMTM interface module 104 may detect this situationby evaluating the number of echoes received within a given time periodand making beam size adjustments accordingly. Once a target is detected,the iMTM interface module 104 determines how to adjust the beam focus.This is achieved by changing the specific configurations and conditionsof the iMTM antenna structure 106. For example, in one scenario thevoltages on the varactors are adjusted. In another scenario, a subset ofiMTM unit cells is configured as a subarray. This configuration meansthat this set may be treated as a single unit, and all the varactors areadjusted similarly. In another scenario, the subarray is changed toinclude a different number of unit cells, where the combination of iMTMunit cells in a subarray may be changed dynamically to adjust toconditions and operation of the system.

All of these detection scenarios, analysis and reactions may be storedin the iMTM interface module 104 and used for later analysis orsimplified reactions. For example, if there is an increase in the echoesreceived at a given time of day or on a specific highway, thatinformation is fed into the antenna controller 110 to assist inproactive preparation and configuration of the iMTM antenna structure106. Additionally, there may be some subarray combinations that performbetter, such as to achieve a desired result, and this is stored in thememory 124.

Attention is now directed at FIG. 2, which shows a schematic diagram ofan iMTM antenna module for use with the iMTM radar system of FIG. 1 inaccordance with various examples. iMTM antenna module 200 has an iMTMantenna structure 202 coupled to an antenna controller 204, a centralprocessor 206, and a transceiver 208. A transmission signal controller210 generates the specific transmission signal, such as an FMCW signal,which is used for radar sensor applications as the transmitted signal ismodulated in frequency, or phase. The FMCW signal enables a radar tomeasure range to a target by measuring the phase differences in phase orfrequency between the transmitted signal and the received or reflectedsignal. Within FMCW formats, there are a variety of modulation patternsthat may be used within FMCW, including triangular, sawtooth,rectangular and so forth, each having advantages and purposes. Forexample, sawtooth modulation may be used for large distances to atarget; a triangular modulation enables use of the Doppler frequency,and so forth.

Other modulation types may be incorporated according to the desiredinformation and specifications of a system and application. For example,the transmission signal controller 210 may also generate a cellularmodulated signal, such as an Orthogonal Frequency Division Multiplexing(“OFDM”) signal. In some examples, the signal is provided to the iMTMantenna module 200 and the transmission signal controller 210 may act asan interface, translator or modulation controller, or otherwise asrequired for the signal to propagate through a transmission line system.The received information is stored in a memory storage unit 212, whereinthe information structure may be determined by the type or transmissionand modulation pattern.

The iMTM antenna structure 202 radiates the signal to a radiating arrayof iMTM cells in the iMTM antenna arrays 216-18. In various examples,the iMTM antenna structure 202 includes a feed distribution module 220,having an impedance matching structure 222 and a reactance controlstructure 224. The reactance control structure 224 includes acapacitance control mechanism controlled by antenna controller 204,which may be used to control the phase of a radiating signal fromradiating array structures, such as iMTM antenna arrays 216-18.

In operation, the antenna controller 204 receives information from othermodules in iMTM antenna module 200 and/or from iMTM interface module 104in FIG. 1 indicating a next radiation beam, wherein a radiation beam maybe specified by parameters such as beam width, transmit angle, transmitdirection and so forth. The antenna controller 204 determines a voltagematrix to apply to the reactance control mechanisms in iMTM antennastructure 202 to achieve a given phase shift or other parameters. Inthese examples, the iMTM antenna structure 202 is adapted to transmit adirectional beam without using digital beam forming methods, but ratherthrough active control of the reactance parameters of the individualiMTM cells that make up each iMTM antenna array 216-18.

Transceiver 208 prepares a signal for transmission, such as a signal fora radar device, wherein the signal is defined by modulation andfrequency. The signal is received by the iMTM antenna structure 202 andthe phase of the iMTM cells in the iMTM antenna arrays 216-18 isadjusted at the direction of the antenna controller 204. In someexamples, transmission signals are received by a portion, orsubarray(s), of the iMTM antenna arrays 216-18 (e.g., subarray 232).These iMTM antenna arrays 216-18 are applicable to many applications,including radar, cellular antennas, and autonomous vehicles to detectand identify targets in the path of or surrounding the vehicle.Alternate examples may use the iMTM antenna arrays 216-18 for wirelesscommunications, medical equipment, sensing, monitoring, and so forth.Each application type incorporates designs and configurations of theelements, structures and modules described herein to accommodate theirneeds and goals.

In iMTM antenna module 200, a signal is specified by antenna controller204, which may be at the direction of an iMTM interface module (e.g.,iMTM interface module 104 in FIG. 1), a sensor fusion module (describedbelow with reference to FIG. 12) via interface to sensor fusion 214, orit may be based on program information from memory storage 212. Thereare a variety of considerations to determine the beam formation, whereinthis information is provided to antenna controller 204 to configure thevarious elements of iMTM antenna arrays 216-18, which are describedherein. The transmission signal controller 210 generates thetransmission signal and provides it to the feed distribution module 220,which then provides it to feed networks 226-28 coupled to iMTM antennaarrays 216-18. Feed networks 226-28 may include a plurality oftransmission lines configured with discontinuities within a conductivematerial.

The feed distribution module 220 has an impedance matching structure 222and a reactance control structure 224 for respectively matching inputsignal parameters with the iMTM cells and providing phase shift controlto each cell. The impedance matching structure 222 may include adirectional coupler having an input port to each of adjacenttransmission lines in the feed networks 226-28. The adjacenttransmission lines in feed networks 226-28 and the impedance matchingstructure 222 may form a super element, wherein an adjacent transmissionline pair has a specific phase difference, such as a 90° phasedifference with respect to each other.

The impedance matching structure 222 works in coordination with thereactance control structure 224 to provide phase shifting of theradiating signal(s) from the iMTM antenna arrays 216-18. In variousexamples, the reactance control structure 224 includes a reactancecontrol mechanism controlled by antenna controller 204, which may beused to control the phase of a radiating signal from the iMTM cells inarrays 216-18 and to adjust the effective reactance of a transmissionline and/or a cell fed by a transmission line in the feed networks226-28. The reactance control structure 224 may, for example, include aphase shift network system (not shown) to provide any desired phaseshift up to 360°. The phase shift network system may include multiplevaractors to achieve the desired phase shift.

One or more reactance control mechanisms may be placed within atransmission line in the feed networks 226-28. Similarly, reactancecontrol mechanisms may be placed within multiple transmission lines orwithin each iMTM radiating cell to achieve a desired result. Thereactance control mechanisms may have individual controls or may have acommon control. In some examples, a modification to a first reactancecontrol mechanism is a function of a modification to a second reactancecontrol mechanism.

The impedance matching element 222 and the reactance control element 224may be positioned within the architecture of feed distribution module220; one or both may be external to the feed distribution module 220 formanufacture or composition as an antenna or radar module. The impedancematching element 222 works in coordination with the reactance controlelement 224 to provide phase shifting of the radiating signal(s) fromiMTM antenna arrays 216-18.

As illustrated, iMTM antenna structure 200 includes the iMTM antennaarrays 216-18, composed of individual iMTM cells such as iMTM cell 230.The iMTM antenna arrays 216-18 may take a variety of forms and aredesigned to operate in coordination with the feed distribution module220, wherein individual iMTM cells correspond to elements within theiMTM transmission arrays 216-18. In various examples, the transmissionsignals sent by the transceiver 208 are received by a portion, orsubarray, of iMTM antenna arrays 216-18 (e.g., subarray 232). Each ofthe iMTM antenna arrays 216-18 is an array of individual iMTM radiatingcells (e.g., an 8×16 array), wherein each of the iMTM cells (e.g., MTMcell 230) has a uniform size and shape; however, some examples mayincorporate different sizes, shapes, configurations and array sizes.

Each iMTM cell (e.g., iMTM cell 230) is an artificially structuredelement used to control and manipulate physical phenomena, such aselectromagnetic (“EM”) properties of a signal including the amplitude,phase, and wavelength. Metamaterial structures behave as derived frominherent properties of their constituent materials, as well as from thegeometrical arrangement of these materials with size and spacing thatare much smaller relative to the scale of spatial variation of typicalapplications. A metamaterial is not a tangible new material, but ratheris a geometric design of known materials, such as conductors, thatbehave in a specific way. An iMTM cell such as cell 230, may be composedof multiple microstrips, gaps, patches, vias, and so forth having abehavior that is the equivalent to a reactance element, such as acombination of series capacitors and shunt inductors. Variousconfigurations, shapes, designs and dimensions are used to implementspecific designs and meet specific constraints. In some examples, thenumber of dimensional freedom determines the characteristics, wherein adevice having a number of edges and discontinuities may model aspecific-type of electrical circuit and behave in a similar manner. Inthis way, an iMTM cell radiates according to its configuration. Changesto the reactance parameters of the iMTM cell change the radiationpattern. Where the radiation pattern is changed to achieve a phasechange or phase shift, the resultant structure is a powerful antenna orradar, as small changes to the iMTM cell result in large changes to thebeamform.

The iMTM cells include a variety of conductive structures and patterns,such that a received transmission signal is radiated therefrom. Invarious examples, each iMTM cell (e.g., cell 230) has some uniqueproperties. These properties may include a negative permittivity andpermeability resulting in a negative refractive index; these structuresare commonly referred to as left-handed materials (“LHM”). The use ofLHM enables behavior not achieved in classical structures and materials,including interesting effects that may be observed in the propagation ofelectromagnetic waves, or transmission signals. Metamaterials can beused for several interesting devices in microwave and terahertzengineering such as antennas, sensors, matching networks, andreflectors, such as in telecommunications, automotive and vehicular,robotic, biomedical, satellite and other applications. For antennas,metamaterials may be built at scales much smaller than the wavelengthsof transmission signals radiated by the metamaterial. Metamaterialproperties come from the engineered and designed structures rather thanfrom the base material forming the structures. Precise shape,dimensions, geometry, size, orientation, arrangement and so forth resultin the smart properties capable of manipulating EM waves by blocking,absorbing, enhancing, or bending waves.

The iMTM antenna arrays 216-18 may have a periodic arrangement (e.g.,array, lattice, etc.) of iMTM cells that are each smaller than thetransmission wavelength. When a transmission signal is provided to theiMTM antenna structure 202, such as through a coaxial cable or otherconnector, the signal propagates through the feed distribution module220 to the iMTM transmission arrays 216-18 for transmission through theair.

Note that as illustrated, there are two iMTM antenna arrays 216-18.However, iMTM antenna structure 202 may incorporate multiple otherantenna arrays. In various examples, each iMTM antenna array may be fortransmission and/or receiving of radiation patterns, where at least oneof the arrays is for transmission in the azimuth, or horizontal,direction, and at least another is for receiving of radiation patternsover the elevation of the array, with the antenna arrays havingorthogonal radiation beams. Note also that the iMTM antenna arrays216-18 are shown with separate feed networks 226-28, but could in someexamples, share a feed network. In various examples, antenna arrays maybe configured to detect different targets, e.g., a set of antenna arraysmay be configured to enhance the detection and identification ofpedestrians, another set of antenna arrays may be configured to enhancethe detection and identification of other vehicles, and so forth. In thecase of pedestrians, the configuration of the antenna arrays may includepower amplifiers to adjust the power of a transmitted signal and/ordifferent polarization modes for different arrays to enhance pedestriandetection.

Referring now to FIG. 3, a flowchart illustrating the operation of anexample iMTM radar system in more detail is described. In one example,the iMTM radar system may be implemented as the iMTM radar system 100 ofFIG. 1. In operation, the antenna controller 110 is responsible fordirecting the iMTM antenna structure 106 to generate RF beams withdetermined parameters such as beam width, transmit angle, etc. (302).The antenna controller 110 may, for example, determine the parameters atthe direction of iMTM interface module 104, which may at any given timewant to focus on a specific area of a FoV upon identifying targets ofinterest in the vehicle's path. The antenna controller 110 determinesthe direction, power, and other parameters of the beams and controls theiMTM antenna structure 106 to achieve beam steering in variousdirections. The antenna controller 110 also determines a voltage matrixto apply to capacitance control mechanisms in the iMTM antenna structure106 (or coupled to the iMTM antenna structure 106) to achieve a givenphase shift. In some examples, the iMTM antenna structure 106 is adaptedto transmit a directional beam through active control of the reactanceparameters of the individual iMTM cells in the iMTM antenna arrays(e.g., arrays 216-18) of the iMTM array structure 106. The iMTMinterface module 102 provides control actions to the antenna controller110 at the direction of the target identification and decision module114, described in more detail below.

Next, the iMTM antenna structure 106 radiates RF beams having thedetermined parameters (304). The RF beams are reflected off of targetsin and around the vehicle's path (e.g., in a 360° field of view) and arereceived by the transceiver module 108 in the iMTM antenna module 102(306). The iMTM antenna module 102 then transmits 4D radar data to thedata pre-processing module 112 for encoding into a point cloud (308).The micro-doppler module 116 coupled to the iMTM antenna module 102 andthe iMTM interface module 104 extracts micro-doppler signals from the 4Dradar data to aid in the identification of targets by the targetidentification and decision module 114 (310). The micro-doppler module116 takes a series of RD maps from the iMTM antenna module 102 andextracts a micro-doppler signal from them. The micro-doppler signalenables a more accurate identification of targets as it providesinformation on the occupancy of a target in various directions.

The target identification and decision module 114 receives the 4D radardata from the iMTM antenna module 102, processes the radar data todetect and identify targets, and determines the control actions to beperformed by the iMTM antenna module 102 based on the detection andidentification of such targets (312). For example, the targetidentification and decision module 114 may detect a cyclist on the pathof the vehicle and direct the iMTM antenna module 102, at theinstruction of its antenna controller 110, to focus additional RF beamsat given phase shift and direction within the portion of the field ofview corresponding to the cyclist's location.

The iMTM interface module 104 also includes a multi-object tracker 118to track the identified targets over time, such as, for example, withthe use of a Kalman filter (314). Information on identified targets overtime are stored at a target list and occupancy map 120, which keepstracks of targets' locations and their movement over time as determinedby the multi-object tracker 118. The tracking information provided bythe multi-object tracker 118 and the micro-doppler signal provided bythe micro-doppler module 116 are combined to produce an outputcontaining the type of target identified, their location, theirvelocity, and so on (316). This information from iMTM interface module104 is then used to determine next actions to be performed by the iMTMantenna module 102 such as what beams to send next and with whichparameters (e.g., beam width, azimuth and elevation angles, etc.) (318).The determination may also include a selection of subarrays in the iMTMantenna arrays in the iMTM antenna module 102 from which to send thenext beams. The output from the iMTM interface module 104 is also sentto a sensor fusion module (described in more detail below with referenceto FIG. 12) where it is processed together with information from othersensors in the vehicle (320).

FIG. 4 illustrates the encoding of 4D radar data into a point cloud inaccordance with various examples. The 4D radar data generated by an iMTMantenna module (e.g., iMTM antenna module 102 of iMTM radar system 100of FIG. 1) can be represented in a 4D hypercube H 402. Each point in thehypercube H corresponds to an intensity value I at a given range r,azimuth angle ϕ, elevation angle θ, and velocity v. The data inhypercube H may be extracted from FMCW radar pulses and contain bothnoise and systematic artifacts from Fourier analysis of the pulses. Inorder to detect and identify targets in the 4D radar data represented byhypercube H in real-time, it is beneficial to encode the hypercube intoa point cloud 404. In some examples, this may be accomplished byreducing the 4D data set into a point cloud by aggregating the 4D dataand extracting values that may correspond to targets for identification.In one example, 4D radar data can be encoded by first noticing that forevery range r, the iMTM radar system may measure multiple velocities(e.g., from reflections returned from the ground and targets orbackground). The 4D radar data is a series of RD maps for multipleangles/beams that can be encoded to isolate the specific ranges, anglesand velocities where targets are identified.

FIG. 5 illustrates an example data encoding to generate a point cloudfrom 4D radar data. In the scenario of FIG. 5, iMTM radar 500 ispositioned in a vehicle ahead of the traffic. There are trees (e.g.,trees 502-06), a deer 508, and other objects such as road signs, roadmarkings, road barriers and so on. The iMTM radar 500 transmits radarbeams towards oncoming traffic as a scan, referred to as a raster scanacross the azimuth. For illustration purposes, the scan is at a singleelevation; however, iMTM radar 500 can transmit scans across theelevation spectrum to capture a 360° FoV.

Each beam gets reflected off as it hits targets. For example, beam (θ)₁is reflected off car 510 and tree 504, beam (θ)₂ is reflected off car512 and tree 506, beam (θ)₃ is reflected off car 514 and deer 508, andso on, continuing across the FoV. Each of the individual beams has acorresponding RD map. The set of “slices” or RD maps 516 represents aset of ranges and velocities for each azimuth angle θ. Some of theslices may not have any meaningful data as the corresponding beam maynot have hit any targets. The first step in the data encoding of the RDdata into a point cloud is to isolate the specific ranges and azimuthangles where targets are present and can be identified. To illustratethis process, consider the mapping 518, where targets are plottedaccording to azimuth angle and range. In the mapping 518, azimuth anglesθ are numbered from 1 to n, corresponding to the ordering of beams fromiMTM radar 500. For beam (θ)_(n) and velocity range r₁, there is amarking 520, which corresponds to target 522 in a first slice of slices516. Each slice may have a range identified wherein a target isdetected. The slices within slices 516 that had no target identified areremoved from the data set to reduce processing.

Note that in the case that multiple contiguous pixels (or voxels)indicate the presence of an object, they are aggregated using prepackedblob image analysis, and the blob center of mass is selected. Anotheroption would be to average the RD maps from all of the highlightedpixels (voxels). The simplest approach, is to simply take the velocitycorresponding to the maximum intensity of return signal. Note also thatwhile there may be multiple objects which differ only by their r, θ, orϕ coordinate, there cannot be two targets which have the same (r, θ, ϕ)but different velocities (since two targets cannot occupy the same spaceat the same time and be differentiated with sub-pixel or sub-voxelprecision). That is, the rest of the velocity information can bediscarded and the encoded RD data is therefore a set with only thevaluable information for processing and target identification.

FIG. 6 illustrates other example data encodings to generate a pointcloud from 4D radar data. In one example, hypercube H 602 is encodedinto two cubes of range, azimuth, and elevation angles. One cuberepresents intensity or brightness levels of reflections from targets(604), and another cube represents velocity information (606). Bothcubes are a way to extract relevant information from the 4D radar dataset and reduce its size for processing. A point cloud can then be formedfrom the relevant data in the cubes, i.e., from the data correspondingto targets of interest.

In another example, the hypercube H can be encoded by implementing anautoencoder 608 or other such neural network on the velocityinformation. Autoencoder 608 is a feed-forward neural network that iscapable of reconstructing an input at the output under certainconstraints. Autoencoders directly learn features from unlabeled data inan unsupervised mode (i.e., by first encoding and then decoding inputs).Using autoencoder 608 in the data pre-processing module 112 improves theperformance of the target identification and decision module 114 andreduces its computational cost. Autoencoder 608 can also be used forinformation other than velocity, such as azimuth and elevation data.

Note that the point cloud generated from hypercube H 602 still encodesmore data than traditional point clouds, such as those used in lidars. Alidar point cloud has data tuples of the form (x, y, z, B), where x, y,and z, are distance coordinates, and B represents the intensity orbrightness at those coordinates. In contrast, the point cloud that isencoded from hypercube H 602 may be thought of as a point cloud withtuples of the form (x, y, z, {right arrow over (B)}), where {right arrowover (B)} is a vector encoding brightness, velocity and angularinformation. It is appreciated that the data encodings illustrated inFIGS. 5-6 are example data encodings; other data encodings may beimplemented to generate a point cloud from a hypercube containing 4Dradar data.

Once the point cloud is generated, the data may be further pre-processedto correct for Non-Line-of-Sight (“NLOS”) information. A point cloudobtained by a radar may include targets in the direct view orLine-of-Sight (“LOS”) of the radar, NLOS targets that are “around thecorners” or hidden from view from the radar, or NLOS reflections of LOStargets due to multi-path propagation of RF waves. The NLOS reflectionsare not actual targets, but rather, they represent an illusion due toreflected waves off of the actual target returning on a path differentthan a direct LOS path. Such illusions may make it difficult toaccurately detect an actual target like another vehicle, a pedestrianand so on, and decrease the reliability of radars in autonomous drivingapplications by increasing the probability of false alarm.

Attention is now directed to FIG. 7, which illustrates a schematicdiagram of a NLOS correction module for use in a data pre-processingmodule (e.g., data pre-processing module 112) in an iMTM radar inaccordance with various examples. NLOS correction module 700 receives aradar point cloud 702 (i.e., a point cloud encoded from a radarhypercube such as illustrated in FIGS. 4-6) and generates a correctedpoint cloud 704 to properly account for NLOS reflections of actual LOStargets and provide an accurate localization of NLOS targets. NLOScorrection module 700 has in essence two tasks to perform: for allpoints in a point cloud S, s_(i)∈S, (1) is s_(i) the result of areflection from a planar reflecting surface? (2) If so, where is thetrue location of the target corresponding to s_(i)?

The first task is performed by Planar Surface Identification Module 706,which locates all significant planar reflecting surfaces in the field ofview of the radar system incorporating NLOS correction module 700. Oncethe plane reflecting surfaces are located, the second task is performedby NLOS Reflection Remapping Module 708, which remaps the NLOSreflections of a target about the identified planar reflecting surfacesto determine a best estimate of its true location.

Note that the Planar Surface Identification Module 706 may also receivea supplemental point cloud 710, e.g., a lidar point cloud, to aid in theidentification of the planar reflecting surfaces. The Planar SurfaceIdentification Module 706 may, for example, identify the planarreflecting surfaces in the supplemental point cloud 710 and then remapthe NLOS reflections in NLOS Reflection Remapping Module 708 in theradar point cloud 702. Alternatively, the identification of the planarreflecting surfaces may be performed with the radar point cloud 702using the supplemental point cloud 710 to verify that the planarreflecting surfaces were located correctly. The vice-versa scenario mayalso be used, with the supplemental point cloud 710 providing the datafor the identification and the radar point cloud 702 providing the datato confirm that the identification is correct. Further, theidentification may be performed in both of point clouds 702 and 710 andthe results may be compared to determine the planar reflecting surfacelocations. It is appreciated that a number of point clouds may be usedin this identification of planar reflecting surfaces by Planar SurfaceIdentification Module 706. The NLOS Reflection Remapping Module 708remaps the NLOS reflections about the identified planar reflectingsurfaces using the radar point cloud 702.

FIG. 8 illustrates the operation of the NLOS correction module of FIG. 7in more detail. NLOS correction module 700 starts out by applying thePlanar Identification Module 706 to a point cloud S (804). The pointcloud may be a radar point cloud such as radar point cloud 802 or asupplemental point cloud. Alternatively, both point clouds may be usedto generate two results that are compared. In various examples, PlanarSurface Identification Module 706 implements a 3D Kernel-Based HoughTransform (“3DKHT”) to detect the planar reflecting surfaces from thepoint cloud. The result of the 3DKHT application to a point cloud S is alist of L candidate planar surfaces with corresponding locations,orientations, and confidence estimates.

Candidate planar surfaces are compared to a confidence brightnessthreshold to indicate when there truly is a significant planar surfacein the field of view. The spurious surfaces, i.e., candidate surfacesthat are below the confidence brightness threshold, are then discarded(806). In general, the cost for false negative results (failing topredict a planar reflecting surface when in fact one exists) is muchlower than the cost for false positives (predicting a reflection wherenone exists). Due to the high cost of false positives, it is likely thatthe confidence brightness threshold may be set high.

With the planar reflecting surfaces now identified, the point cloud S istransformed into a spherical coordinate system centered on the radaritself (808). The angular space of the point cloud S, i.e., the azimuthand elevation angles (ϕ,θ), is discretized into k² bins (810). For eachof the L planar surfaces, NLOS correction module 700 proceeds to extractthe bins that the planar surface intersects (812). The planar surface'sposition and its surface normal vector are also extracted (814). If twoplanar surfaces intersect the same bin, the more distant surface isignored. For discussion and illustration purposes, consider that the Lplanar surfaces intersect M bins. The surface positions of theidentified L planar surfaces in each bin intersection and their surfacenormal vector define M different reflection operations about therelevant surfaces (816). For each affected bin, the coordinates of thepoints in S whose distance from the radar exceeds the distance from theradar to the intersecting plane are then remapped by a reflection aboutthe intersecting plane to locate the targets (818).

Note that this reflection operation can be defined in O(1) for each binand performed in O(n) where n is the number of points to be reflected.Since each bin is expected to have on average N/k² points, and MαLk²,the entire reflection operation is expected to scale as

$\frac{LMN}{k^{2}} \cong {{LN}.}$If the confidence brightness threshold is kept high, there will not bean enormous number of planar surfaces, and so this scaling will be fine.Note also that the 3DKHT implementation for the Planar SurfaceIdentification Module 706 is a deterministic method of planar Houghtransformation which runs in N log N. The 3DKHT implementation has lowenough computational and memory cost to be feasible on inexpensivehardware in real time. It is appreciated that other implementations foridentifying planar reflecting surfaces may also be used by PlanarSurface Identification Module 706.

It is also appreciated that there may a fair amount of trial and errorin determining the proper confidence brightness threshold. One approachis to simplify the planar identification by looking first for horizontalplanes. Further accuracy can be obtained by filtering out points due totargets with a non-zero velocity relative to a road, since theydefinitely do not correspond to a fixed planar surface. Suchimplementation may be used for example to image the back of a vehicletwo places ahead of the autonomous driving vehicle in a line of cars, orimage vehicles moving behind a line of stopped cars.

After the data is pre-processed to encode it into a point cloud andgenerate a NLOS-corrected point cloud, the NLOS-corrected point cloud isinput into the iMTM interface module 104 for target detection andidentification. Attention is now directed to FIG. 9, which shows aschematic diagram of an iMTM interface module of FIG. 1 in accordancewith various examples. iMTM interface module 900 has two neuralnetworks: a deep convolutional neural network (“CNN”) in targetdetection module 902 and a decision network 904. CNN 902 takes in theNLOS-corrected point cloud 906 and provides output data detectingtargets, identifying them (e.g., whether a vehicle, pedestrian, cyclist,wall, etc.), their location, velocity, and other identifyinginformation. Decision network 904 is a Q-learning network that receivesthe output data from CNN 902 and determines an action for the iMTMantenna module 102 to perform, such as, for example, to steer RF beamsto a given direction in the field of view.

In various examples, CNN 902 is a fully convolutional neural network(“FCN”) with three stacked convolutional layers from input to output(additional layers may also be included in CNN 902). Each of theselayers also performs the rectified linear activation function and batchnormalization as a substitute for traditional L2 regularization andincludes three filters. As a preliminary step to processing the pointcloud 906, point cloud 906 is run through a dynamic threshold. Doing soencodes much higher resolution radar data while still retainingcomputational efficiency. Targets are shown in the point cloud 906 asvoxels, which are values in the multi-dimensional space of the radardata containing range, velocity, azimuth and elevation angles.

The CNN 902 uses small regions of a visual field and identifies edgesand orientations in the field, much like a filter for an image. Theimage goes through a series of convolutional, nonlinear sampling throughlayers, resulting in a probability. The layers include a convolutionallayer that looks at these small regions individually, referred to asreceptive fields. The filter process incorporates weights in connectionsbetween layers, and when the original information is passed through thislayer, the result is a reduced set of data, referred to as a featuremap. The feature map identifies targets detected in each receptivefield. Note that there may be any number of feature maps as a functionof features used in processing. The layers of the CNN 902 detect a firstlevel of features, such as edges. The output of each layer feeds thenext layer, which detects a second level of feature, such as a square.At the output of each layer in CNN 902 is a feature map identifying thelocations of those features. And as data processes through CNN 902, thelayers become more complex to further refine the specific target untilthe target can be properly identified (e.g., as a pedestrian, cyclist,animal, wall, vehicle, etc.). The final layer of the CNN 902 is a fullyconnected layer that takes an input feature map and outputs anN-dimensional vector, where N is the number of features or classes. Eachnumber of the N-dimensional vector identifies the probability of eachcorresponding feature.

It is noted that CNN 902 may incorporate other information to help itidentify targets in the vehicle's path and surrounding environment. Forexample, when a target is moving slowly and outside of a road line, itis likely that the target may be a pedestrian, animal, cyclist, and soon. Similarly, when a target is moving at a high speed, but lower thanthe average speed of other vehicles on a highway, CNN 902 may use thisinformation to determine if the target is a bus or a truck, which tendin general to move more slowly. The location of a target, such as in thefar-right lane of a highway, may also provide an indication as towhether the target may be a slower-moving type of vehicle. If themovement of the target does not follow the path of a road, then thetarget may be an animal, such as a deer crossing the road. All of thisinformation may be determined from a variety of sensors and otherinformation available to the vehicle, including information providedfrom weather and traffic services, other vehicles or the environmentitself, such as smart roads and smart traffic signals. A sensor fusionmodule (described below with reference to FIG. 12) analyzes all theinformation available from the sensors to more accurately detect andidentify each target.

The operational accuracy of the CNN 902 is determined by severalfactors, and one is the training process that provides feedback to thenetwork to adjust its weights; this process is called backpropagation. Aflowchart illustrating the steps for training the CNN 902 is shown inFIG. 10. The CNN 902 trains on known sets of input-to-output data. Forexample, an input may be the camera data received from a camera sensorat a time ti. The known input-output dataset is selected as either rawdata or may be synthetic data; the data is digitized, and specificparameters extracted (1002). The data may also be compressed orpre-processed. Either way, there is a set of input data received from asensor (e.g., iMTM antenna module 102). The CNN 902 does a forward passthrough each one of its layers, computing each layer output based on theweights in the layer, and passing the output to the next layer (1004).The output data of CNN 902 is then what information you would like theCNN 902 to provide you when it receives this set of sensor data, i.e.,the output of CNN 902 will be in the same form as the known output ofthe selected data. Its value, however, may differ from the known output.The next step is to compare the output of CNN 902 with the known,expected output from the selected dataset (1006). This can beimplemented in a number of ways, such as by Euclidean distance, crossentropy, weighted cross entropy, and other such measures.

A score 908 is determined as an indication of how close the output ofCNN 902 matches the expected output (1008). Steps 1004-1008 iterateuntil the scores indicate that the network is trained (1010), that is,until an error tolerance for the scores is small enough and the outputsof CNN 902 given the known inputs are within a desired tolerance fromthe known outputs. If they are not, then the score 908 is sent back tothe CNN 902 to adjust its weight (1012) and steps 1004-1008 continue toiterate. Training of CNN 902 is therefore an iterative process, whichterminates when the output of the network is sufficiently close to thedesired results. There are a variety of methods to adjust the weights inthe CNN 902; the goal is to have a CNN 902 that can receive any sensorinformation (e.g., point cloud 906) and predict the targets andenvironment as closely as possible.

In various examples, the CNN 902 may be trained on one type of data(e.g., lidar point cloud data, radar synthetic data, etc.) and thenretrained (1014) to adapt to a new set of data (e.g., radar data).Retraining may be done using a combination of synthesized data and realsensor data. Real sensor data may be labeled with labels 910, which are,for example, bounding boxes placed around known items in view in eachmulti-dimensional slice of the radar data. Note that labels 910 fortraining CNN 902 may not be necessary, such as when an autoencoder 606is used in the data pre-processing module 112.

As shown in FIG. 9 and described above, the output of CNN 902 is sent toDNN 904 so that DNN 904 can determine an action for the iMTM antennamodule 102 to perform, such as, for example, to steer RF beams to agiven direction in the FoV. In order to select the best action, DNN 904is trained based on reinforcement learning, a machine learning techniqueinspired by behavioral psychology. The idea is to have DNN 904 choose anaction for a given state such that its reward is maximized. In thiscase, the state is the output of the CNN 902, the action is a selectionof beam parameters for the iMTM antenna module 102 to know where todirect its next beams with the selected parameters (e.g., beam width,direction, etc.), and the reward is the performance of the DNN 904following the selections.

Training the DNN 904 boils down to indicating to the DNN 904 when it isdoing well and when it is doing poorly. For example, reinforcementlearning is used to train dogs. You cannot tell the dog what to do, butover time the dog will understand that certain actions lead to morerewards. The rewards are also not the same; some rewards may be morelikely or desirable than others. The goal of DNN 904 is then to maximizeits expected future reward of an action given a state. Training of DNN904 is accomplished by teaching the DNN 904 to have the optimalrepresentation of the space of states, actions, and rewards.

Attention is now directed to FIG. 11, which illustrates a flowchart fortraining the DNN 904 of FIG. 9. The first step in training 1100 is toprepare a known input-output training dataset (1102). The dataset caninclude synthetic and/or real data output by a radar system such as theiMTM radar system 100 of FIG. 1. As described above, the data from theseradar systems is multi-dimensional and includes measures such as range,velocity, azimuth and elevation for each beam. In training DNN 904, afull 4D data set can be used, or alternatively, DNN 904 may be trainedwith a smaller dimensional set. In one example, the dataset used intraining is a 4D hybercube; in other examples, a 3D data cube is used byscanning data at a fixed elevation (e.g., zero or other elevation) andrecording the range, velocity and azimuth angles.

Let this dataset be represented by a cube or hypercube denoted by M. Ineach orientation of a beam, a sequence of pulses is collected to containsufficient information to fill one slice of M. This dataset may bereferred to as the “raw data cube,” as it contains information which maybe preprocessed, but has not yet been fed to any machine learningcomponents. Out of this dataset, a set of k directions is selected. Theselection may be performed randomly or in other ways. Each of the kdirections is associated with known outputs. That is, the dataset may beprepared by generating beams in the radar system in the k directions ina road-like environment, recording the reflections from known targets,and labeling the data with bounding boxes around the targets so thateach target's location and type (e.g., vehicle, wall, pedestrian,animal, cyclist, etc.) is known. Alternatively, the dataset may containa set of known input-output pairs representing a real-world scenario ofa vehicle in a road.

The raw data cube containing data corresponding to these k directions isthen fed through the CNN 902 (1104). The output of CNN 902, which mayhave already been trained, is compared with the known output from theselected dataset (1106). A score is computed based on the comparison(1108). In various examples, a single score may be computed for eachdirection; in other examples, a composite score may be computed for thek directions. The output of the CNN 902 is input into the DNN 904(1110). The DNN 904 also has a set of experience data tuples of (state,action, reward, next-state) (1112). The state, as described above,corresponds to the output of the CNN 902, the action corresponds to aselected set of beam parameters, and the reward is a desired performancemeasure following the selections. In various examples, the reward may bea function such as:

$\begin{matrix}{r \propto {\ln\left( \frac{{loss}_{i}}{{loss}_{i - 1}} \right)}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$where loss may be a measure such as Euclidean distance, weighted binarycross entropy, or another such measure. Note that loss is not useddirectly, so as not to punish good actions taken in bad situations andvice-versa.

During training, DNN 904 is run to explore the action space with a fixedprobability of taking random actions. Each experience data tuple is thenrecorded as it's encountered and stored in a buffer of fixed length(e.g., of a length of 10⁵). DNN 904 is trained by sampling mini-batchesrandomly from this buffer and computing a state-action Q function knownin Q-learning as the Bellman equation:Q(s,a)=r+γ max_(a′) Q(s′,a′)  (Eq. 2)where γ is a discount rate for the rewards between 0 and 1 to take intoaccount the fact that not all rewards are the same: the larger the valueof γ, the smaller the discount (i.e., DNN 904 cares more about thelong-term reward), and the smaller the γ, the bigger the discount (i.e.,DNN 904 cares more about the short-term reward). Conceptually, Eq. 2states that the maximum future reward for state s and action a is theimmediate reward r plus the maximum future reward for the next state.The Q function may be implemented as the buffer, with states as rows andactions as columns. In various examples, for computing the maximumfuture reward for the next state (i.e., γ max_(a′)Q(s′,a′)), previous,frozen versions of DNN 904 are used to avoid instabilities andoscillations during training. Furthermore, because DNN 904 is expectedto require less forward planning than many Q-learning applications, thediscount rate γ is initially set to 0 and gradually increased duringtraining. This technique helps the network converge more rapidly and hasnot been introduced in the Q-learning literature.

Based on Eqs. 1-2, DNN 904 computes a score for every possible action(1114). In deterministic operation, the k highest scoring options areselected (1116) and the dataset is updated with data corresponding tothe selected actions (1118). The updated data set is fed to the CNN 902(1120), which once again produces a score based on the comparisonbetween the output of CNN 902 and the known, expected output from thedataset (1122). This score is compared to the previous score, and basedon this comparison, it is determined that the k selections made by DNN904 were either good or bad (1124). Depending on this determination, DNN904 may be considered to be trained (1126), but if not, its parametersare updated (1128), and training continues with further data.

It is appreciated that while the operation of DNN 904 may occur at aframerate that may require selecting more than one beam at a time,during training the space of actions may be restricted to the selectionof a single beam. This is because it is desired to attribute a change inscore to a particular action, rather than an average score to a group ofactions. To match the framerate goals, the simulated world is frozen fork steps before advancing, so that the effect is to select k beams duringeach timestep as will be done during inference with DNN 904.

It is also appreciated that an additional training mode may be enabled:alternate or simultaneous training of DNN 904 and CNN 902.Alternatively, CNN 902 and DNN 904 may be first trained with one type ofdata (e.g., lidar data) and retrained with radar data. The networks CNN902 and DNN 904 may also be trained with real, labelled data in areal-world subsampling scenario. In this case, rather than being able tochoose any of the possible actions, the action space may be restrictedto the subset of actions that were actually taken. Having selected oneof these actions, training proceeds as before. If done entirelyasynchronously, this constitutes an “off-policy” approach. However, thisprocess may be iterated a number of times, where each new dataset iscollected using the latest policy network. Note that when an autoencodersuch as autoencoder 606 of FIG. 6 is used to pre-process the radar dataprior to it being fed through CNN 902 and DNN 904, the training processis improved—both in its performance and computational complexity.

Returning to FIG. 9, the output of the CNN 902 and DNN 904 is fedthrough a multi-object tracker 912 to track the identified targets overtime, such as, for example, with the use of a Kalman filter. Informationon identified targets over time are stored at a target list andoccupancy map 914, which keeps tracks of targets' locations and theirmovement over time as determined by the multi-object tracker 912. Thetracking information provided by the multi-object tracker 912 and themicro-doppler signal provided by the micro-doppler module 116 of FIG. 1are combined to produce an output containing the type of targetidentified, their location, their velocity, and so on.

The beam control module 916 receives the output from the target list andoccupancy map 914 and determines the adjustments, if any, to be made. Insome examples, the iMTM radar 100 scan begins with a coarse scan havinga large bandwidth. On target detection, the beam width narrows. The beamcontrol module 916 may vary the beam width as quickly or slowly asdesired. In some examples, the beam width is a binary value, and inothers it may take on continuous values. The beam control module 916also instructs the iMTM antenna module 102 where to direct the nextbeam, such as from a specific subarray or subarrays. The beam controlmodule 916 also determines parameters and dimensions of the next beamsfor iMTM antenna module 102. In various examples, the iMTM interfacemodule 900 also includes FoV composite data 918 and memory 920. FoVcomposite data 918 stores information that describes a FoV and memory920 stores useful data for the iMTM radar system, such as, for example,information on which subarrays of the iMTM antenna structure performbetter under different conditions. The beam control module 916 may usethe FoV information stored in FoV composite data 918 and the subarrayinformation stored in memory 920 to better control the parameters of thenext beams.

The target identification information from iMTM radar system 100 is sentto a sensor fusion module, where it is processed together with targetdetection and identification from other sensors in the vehicle. FIG. 12illustrates a schematic diagram of an autonomous driving system havingan iMTM radar in accordance with various examples. Autonomous drivingsystem 1200 is a system for use in a vehicle that provides some or fullautomation of driving functions. The driving functions may include, forexample, steering, accelerating, braking, and monitoring the surroundingenvironment and driving conditions to respond to events, such aschanging lanes or speed when needed to avoid traffic, crossingpedestrians, animals, and so on. The autonomous driving system 1200includes an iMTM radar 1202 and other sensor systems such as camera1204, lidar 1206, infrastructure sensors 1208, environmental sensors1210, operational sensors 1212, user preference sensors 1214, and othersensors 1216. Autonomous driving system 1200 also includes acommunications module 1218, a sensor fusion module 1220, a systemcontroller 1222 and a system memory 1224. It is appreciated that thisconfiguration of autonomous driving system 1200 is an exampleconfiguration and not meant to be limiting to the specific structureillustrated in FIG. 12. Additional systems and modules not shown in FIG.12 may be included in autonomous driving system 1200.

iMTM radar 1202 includes an iMTM antenna module (e.g., iMTM antennamodule 102) for providing dynamically controllable and steerable beamsthat can focus on one or multiple portions of a 360° FoV of the vehicle.The beams radiated from the iMTM antenna module are reflected back fromtargets in the vehicle's path and surrounding environment and receivedand processed by the iMTM radar 1202 to detect and identify the targets.The iMTM radar 1202 also has an iMTM interface module (e.g., iMTMinterface module 104 of FIG. 1) that is trained to detect and identifytargets and control the iMTM antenna module as desired.

Camera sensor 1204 may be used to detect visible targets and conditionsand to assist in the performance of various functions. The lidar sensor1206 can also be used to detect targets outside the vehicle and providethis information to adjust control of the vehicle. This information mayinclude information such as congestion on a highway, road conditions,and other conditions that would impact the sensors, actions oroperations of the vehicle. Camera sensors are currently used in ADASsystems to assist drivers in driving functions such as parking (e.g., inrear view cameras). Cameras are able to capture texture, color andcontrast information at a high level of detail, but similar to the humaneye, they are susceptible to adverse weather conditions and variationsin lighting. Lidar sensors measure the distance to an object bycalculating the time taken by a pulse of light to travel to an objectand back to the sensor. When positioned on top of a vehicle, lidars areable to provide a 360° 3D view of the surrounding environment. However,lidar sensors are still prohibitively expensive, bulky in size,sensitive to weather conditions and are limited to short ranges(typically <200 m), with resolution decreasing with range. Radars, onthe other hand, have been used in vehicles for many years and operate inall-weather conditions. Radars also use far less processing than theother types of sensors and have the advantage of detecting targetsbehind obstacles and determining the speed of moving targets.

Infrastructure sensors 1208 may provide information from infrastructurewhile driving, such as from a smart road configuration, bill boardinformation, traffic alerts and indicators, including traffic lights,stop signs, traffic warnings, and so forth. This is a growing area, andthe uses and capabilities derived from this information are immense.Environmental sensors 1210 detect various conditions outside, such astemperature, humidity, fog, visibility, precipitation, and so forth.Operational sensors 1212 provide information about the functionaloperation of the vehicle. This may be tire pressure, fuel levels, brakewear, and so forth. The user preference sensors 1214 may be configuredto detect conditions that are part of a user preference. This may betemperature adjustments, smart window shading, and so forth. Othersensors 1216 may include additional sensors for monitoring conditions inand around the vehicle.

In various examples, the sensor fusion module 1220 optimizes thesevarious functions to provide an approximately comprehensive view of thevehicle and environments. Many types of sensors may be controlled by thesensor fusion module 1220. These sensors may coordinate with each otherto share information and consider the impact of one control action onanother system. In one example, in a congested driving condition, anoise detection module (not shown) may identify that there are multipleradar signals that may interfere with the vehicle. This information maybe used by the iMTM interface module in system 1200 to adjust the beamsize of the iMTM antenna module so as to avoid these other signals andminimize interference.

In another example, environmental sensor 1210 may detect that theweather is changing, and visibility is decreasing. In this situation,the sensor fusion module 1220 may determine to configure the othersensors to improve the ability of the vehicle to navigate in these newconditions. The configuration may include turning off camera or lasersensors 1204-1206 or reducing the sampling rate of thesevisibility-based sensors. This effectively places reliance on thesensor(s) adapted for the current situation. In response, the iMTMinterface module (e.g., iMTM interface module 104 of FIG. 1) configuresthe iMTM radar 1202 for these conditions as well. For example, the iMTMradar 1202 may reduce the beam width to provide a more focused beam, andthus a finer sensing capability.

In various examples, the sensor fusion module 1220 may send a directcontrol to the iMTM antenna module (e.g., iMTM antenna module 102) basedon historical conditions and controls. The sensor fusion module 1220 mayalso use some of the sensors within system 1200 to act as feedback orcalibration for the other sensors. In this way, an operational sensor1212 may provide feedback to the iMTM interface module and/or the sensorfusion module 1220 to create templates, patterns and control scenarios.These are based on successful actions or may be based on poor results,where the sensor fusion module 1220 learns from past actions.

Data from sensors 1202-1216 may be combined in sensor fusion module 1220to improve the target detection and identification performance ofautonomous driving system 1200. Sensor fusion module 1220 may itself becontrolled by system controller 1222, which may also interact with andcontrol other modules and systems in the vehicle. For example, systemcontroller 1222 may turn the different sensors 1202-1216 on and off asdesired, or provide instructions to the vehicle to stop upon identifyinga driving hazard (e.g., deer, pedestrian, cyclist, or another vehiclesuddenly appearing in the vehicle's path, flying debris, etc.)

All modules and systems in autonomous driving system 1200 communicatewith each other through communication module 1218. Autonomous drivingsystem 1200 also includes system memory 1224, which may storeinformation and data (e.g., static and dynamic data) used for operationof system 1200 and the vehicle using system 1200. Communication module1218 may also be used for communication with other vehicles, referred toas V2V communication. V2V communications may include information fromother vehicles that is invisible to the user, driver, or rider of thevehicle, and may help vehicles coordinate to avoid an accident.

Attention is now directed to FIGS. 13-14, which illustrate processesimplemented in the sensor fusion module 1220 of FIG. 12, and actionsbased on sensor readings. In FIG. 13, a process 1300 looks to see if asignal is received from any of the sensors within a system (1302), suchas in sensor fusion module 1220 of FIG. 12. If no signal is received,processing continues to listen for sensor signals. When a signal isreceived (1302), the sensor fusion module 1220 determines the sensorparameters (1304), which include the information type received from thesensor. This information may be stored for analysis as to actions takenby the vehicle to enable intelligent, flexible, and dynamic control. Theprocess 1300 then continues to compare the signal received to datastored by the sensor fusion module 1220 (1306) wherein such data may bestored in memory (not shown) or stored in a networked repository, suchas a cloud database and system (not shown). At this point, if a controlaction is indicated at 1308, processing continues to determine if thiscontrol action and/or the information received from the sensor willprovide early detection for this or another action. This early detectioncheck (1310) allows the entire sensor ecosystem to take advantage ofinformation from any of the sensors in the autonomous driving system1200. If the sensor information may be used for early detection (1310)then the information is sent to one or more modules (1312), or is storedin memory as a data point in the current scenario. The autonomousdriving system 1200 then takes the indicated action (1314) and returnsto listen for signals at 1302. If the information is not used for earlydetection at 1310, then processing continues to take the indicatedaction at 1314. If no control action is indicated at 1308, processingreturns to listen for sensor signals.

FIG. 14 illustrates another process 1400 according to some examples,wherein the sensor fusion module 1220 configures sensors and controlsfor operation at 1402. This may be a dynamic step or may be a persistentconfiguration. When a target is detected by the iMTM radar 1202 (1404),the process 1400 uses that information to calculate or determinespecifics relating to the target with respect to the antenna position.The angle of arrival (“AoA”) is compared to the transmission angle or ismapped to a subarray in the iMTM antenna structure in iMTM radar 1202(1406). This information is used to determine the position of thedetected target in 2D or 3D space (1408). The range, or distance fromthe antenna to the target, is a function of the radar chip delay (1410).The information from the iMTM radar 1202 and other sensors is used todetermine a silhouette and/or footprint of the target (1412).Optionally, information from the sensor(s) may provide a targetsignature of the target (1414), depending on the target's composition(e.g., metal, human, animal) and so forth. This may be an indication ofthe reflectivity of the target. The target signature is a more detailedunderstanding of the target, which may give dimensions, weight, and soforth. Alternatively, the target may be identified as described abovewith the target identification and decision module 114 of FIG. 1. Thesensor fusion module 1220 then accesses sensor information to determinea control action (1416) and instructs the vehicle to take action (1418).

A variety of information is determined from the iMTM radar 1202; suchinformation may be a function of the modulation waveform and technique,the frequency, the chirp delay, the frequency change of the receivedsignal and so forth. The specific radiation pattern used may be craftedto accomplish specific goals according to the application. The sensorfusion module 1220 enables such control to optimize the system andreduce the processing required. For example, the iMTM radar 1202 may beused to reduce the number of sensors and/or the active time of eachsensor. In this way, some sensors may be disabled during certainconditions, and activated on a change in that condition.

The various examples described herein support autonomous driving withimproved sensor performance, all-weather/all-condition detection,advanced decision-making algorithms and interaction with other sensorsthrough sensor fusion. These configurations optimize the use of radarsensors, as radar is not inhibited by weather conditions in manyapplications, such as for self-driving cars. The ability to captureenvironmental information early aids control of a vehicle, allowinganticipation of hazards and changing conditions. Sensor performance isalso enhanced with these structures, enabling long-range and short-rangevisibility. In an automotive application, short-range is consideredwithin 30 meters of a vehicle, such as to detect a person in a crosswalk directly in front of the vehicle; and long-range is considered to250 meters or more, such as to detect approaching cars on a highway.These examples provide automotive radars capable of reconstructing theworld around them and are effectively a radar “digital eye,” having true3D vision and capable of human-like interpretation of the world.

These capabilities in a radar as iMTM radar 100 of FIG. 1 are enabledwith the use of an iMTM antenna structure such as iMTM antenna structure106 in iMTM antenna module 102. FIG. 15 illustrates an iMTM antennastructure 1500 (or a portion of a structure) having a plurality of iMTMcells arranged in an array of N×N unit cells, wherein for clarity anddiscussion herein each unit cell is identified by a row, column index(i,j). The array can be an asymmetric N×M array as well. For simplicity,a symmetric N×N case is described. For example, from the viewer'sperspective, the unit cell in the upper corner is identified as 1502(1,1); and the unit cell in the bottom right is identified as 1502(N,N). Other configurations are possible based on the application,structure, physics and goals of the antenna structure 1500. Antennastructure 1500 is part of an antenna system or module, e.g., iMTMantenna module 102 of FIG. 1, that includes other modules, some of whichare not shown in this drawing. Similarly, the specific shape of the unitcells may take on any of a variety of shapes that result in thecharacteristics and behavior of metamaterials and are not restricted tosquare or rectangular or any other regular shape.

Each of the unit cells 1502 (i,j) in the antenna structure 1500 mayoperate individually or as part of a subarray. As illustrated, the iMTMinterface module 1512 (e.g., implemented like the iMTM interface module104 of FIG. 1) has associated or grouped specific unit cells intosub-arrays 1504-1510. The iMTM interface module 1512 determines wherethe radiated beam is to be directed, the shape of the beam and thedimensions of the beam. The beam may be a coarse or large bandwidthbeam, a midsized beam or a small, narrow bandwidth beam depending on thesituation, the target detected and the timing of the detection, as wellas other considerations. The iMTM interface module 1512 may preconfigureone or more of the subarrays to anticipate a next action, or may use adefault configuration, such as to start with a broad bandwidth whichenables a faster scan capability or sweep time. For each sweep, the FoVis divided into portions, which may have consistent dimensions,different dimensions or may be dynamically adjusted. In some examples,the iMTM interface module 1512 selects specific directions to have anarrow beam, such as directly in front of the vehicle; other directions,such as on the edges of the FoV may be scanned with a wide beam. Theseand other design considerations are made by the designer in setting upthe iMTM interface module 1512, wherein some are flexible andconfigurable. In the illustrated example, the iMTM antenna structure1500 has several subarrays that are intended to direct the beam and formthe desired radiation pattern.

Once a target is detected and identified, the FoV-to-MTM mapping 1514identifies the portion of the FoV for the iMTM interface module 1512 andmaps that location to a specific iMTM unit cell or subarray that willfocus on and capture more information about the target. In someexamples, the iMTM interface module 1512 has access to various scenariosand may use detected information to predict future conditions on theroad. For example, if the iMTM antenna structure 1500 detects a deerrunning across the road in an area having a known deer path, the iMTMinterface module 1512 may predict the direction of the deer, as well asanticipate other deer that may follow. The radiation beams from antennastructure 1500 may sweep across the FoV, wherein the visual field ofview and the antenna field of view are not necessarily the same. In thiscase, the antenna FoV may be a 2D view, whereas targets are typically3D. Various systems and configurations enable 3D target detection andclassification through placement of transmit and receive antenna arraysand or combinations of multiple transmit to multiple receive structures.

FIG. 16 illustrates an iMTM antenna array 1600 having at least onesub-array 1602 activated to generate beams to capture a specific area orFoV 1612, corresponding to the iMTM radar system 100 of FIG. 1. When thecar 1618 is detected within an area 1616, the iMTM interface module 1606identifies the associated portion 1616 of the FoV 1612. This is mappedto the portion of the iMTM antenna array 1600 that will generate afocused beam in that area; and that portion is sub-array 1602.Similarly, car 1614 is also identified within FoV 1612 in another area;street lamp 1610 and person 1608 are also located within FoV 1612. Theradar system has a mapping from the FoV to the iMTM array 1604, whichmay be configured as a Look Up Table (“LUT”), as a formula, or asanother mapping format that configures subarrays of the iMTM array 1600to generate a beam toward individual portions of the FoV 1612. In thisway, there is low latency dynamic adjustment of the radiation beam forbeam forming and beam steering. The ability to capture multiple targetswith a single subarray acts to further reduce the delay in detection andcommunication, reducing the time from detection to action.

As illustrated in FIG. 16, the mapping between the iMTM antenna array1600 and the FoV 1612 is provided by FoV-to-MTM mapping unit 1604, whichincludes various entries for such correlation. This type of mappingformat may be dynamically adjusted to keep pace with the movement ofvehicles; in addition, this information may be stored in a relationaldatabase or other device to assist the iMTM interface module 1606 inlearning and improving over time. In this way and as described above,the iMTM interface module 1606 may use AI, machine learning, deeplearning, an expert system, and/or other technology to improveperformance of the iMTM radar system for target detection andidentification.

As a vehicle travels, there are different FoV snapshots or slices, suchas from a near-field to a far-field slice. From the perspective of avehicle, there is a near-field FoV, a far-field FoV, and severalmid-field FoVs, which may each be considered as a slice of information.The information may be stored according to angle of arrival, range tothe target, velocity of the target, Doppler information from thereceived signal and so forth. In various examples and as illustrated inFIG. 5, these are referred to as Range-Doppler maps. Each slicecorresponds to an instant in time as the car travels. The iMTM interfacemodule 1606 determines which type of beam is broadcast for each FoV as afunction of many parameters, including, for example, the speed of thecar and the speed of a detected object in relation to the car. The iMTMinterface module 1606 may also determine that for specific conditions,the beams are meant to reach a specific FoV, such as where the car ismoving slowly, a given FoV may be sufficient, but if the car is movingrapidly, then there is a desire to reach a full FoV. Weather conditionswill have an impact as well, such that if the car will take longer toreact, stop or otherwise change the current driving conditions, then theiMTM interface module 1606 may desire to reach the longest FoV to allowthe car time to react. This may be utilized for snow or icy conditions,which dramatically impact how quickly a car may decelerate and/or halt.

Some other considerations for antenna applications, such as for radarantennas used in vehicles, include the antenna design, capabilities, andreceiver and transmitter configurations. A typical electronic systemwith an antenna array consists of two or more antenna elements, a beamforming network, and a receiver and/or transmitter. The beamformingnetwork may consist of a Butler matrix or other antenna arrays combinedwith phase shifting elements. Many different antenna configurations canbe utilized as an antenna element in the antenna array: simple dipole,monopole, printed patch design, Yagi antenna, and so forth. One of theprimary goals for antennas mounted on/in the car is to achieve a compactand aesthetic design. Other goals relate to the type of communicationsignal used for the radar beam. One type of modulation is the FMCWmodulation, which is effective in radar applications, as radar does notneed to pulse, but rather transmits continuously. FMCW is a continuouscarrier modulated waveform that is transmitted as a continuous periodicfunction, such as sinusoid, sawtooth, triangular and so forth. The sweeptime, or sweep period, T_(s), is the time for transmission of one periodof the waveform. The signal transmitted during one sweep period isreferred to as a chirp. There is a difference in the frequency of thetransmit and receive signals that is referred to as the beat frequency,b_(f). The range of the antenna, r, is the distance from the antenna toa detected target, and is a function of the sweep period, beatfrequency, the speed of light, c, and the sweep bandwidth, B_(s). Amoving target induces a Doppler frequency shift that enables radar todetect the relative velocity of the target with respect to the antenna.The phase difference between the transmit and receive signals provideslocation information, while the frequency shift identifies a speed. Inthe case of moving targets, the signal phase distortions may impact theperformance of the antenna array. One way to offset such distortion isto use multiple subarrays at the transmit and receive sides to filterout these impurities. Another way is to adjust the antenna calibrationon-the-fly to reduce the phase distortion of moving targets.

Traditional phase shifting may be used to control the beam of anantenna. Phased array antennas have multiple elements that are fed so asto have a variable phase or time-delay at each element and so that thebeam scans from different angles. The multiple elements provideradiation patterns with lower sidelobes and enables careful beamshaping. The beam can be repositioned for more directed and efficientoperation.

The various examples described herein provide an iMTM antenna structurethat provides phase shifting without the active elements required tochange the phase, or in the traditional ways. The iMTM antennastructures of various examples use the characteristics of themetamaterial shape and configuration to provide phase shifts without theuse of mechanical or electrical phase shifters.

The iMTM antenna arrays described herein may be fed by a variety ofconfigurations, such as a probe feed or a substrate integrated waveguideand so forth. In one example of an iMTM antenna structure 1700,illustrated in FIG. 17, a signal source is provided as a probe 1704,which may be coupled to a ground plane 1702. The probe 1704 supplies thesource signal for the antenna 1700 to generate a modulated EM waveform.A second layer 1706 is positioned over the ground plane 1702. The secondlayer 1706 is made of a dielectric material and has an antenna structure1708 configured thereon. This antenna 1708 is designed to receive thesource signal and generate a relatively flat wave front to meet the iMTMlayer 1710. The antenna 1708 may be a dipole antenna or any otherantenna capable of generating a relatively uniform and flat wave frontacross the entirety of the second layer 1706. The ability to provide thesignal to the iMTM array or to individual subarrays and/or individualunit cells, enables the iMTM antenna 1700 to radiate EM beamforms thatare steerable. The iMTM unit cells are controlled by changes to thereactance behavior of the iMTM unit cells, such as through a variablecapacitor or varactor(s) within each iMTM cell.

Another example is illustrated in FIG. 18, which is a two-layer, probefed iMTM antenna structure 1800. As in the example of FIG. 17, a probe1804 supplies the signal to a ground plane layer 1802. In this example,an iMTM antenna array 1806 is placed over the ground plane with nomiddle layer. The source signal is distributed across the ground plane1802 such that a relatively flat wave form is presented to the iMTMantenna array 1806. The iMTM antenna array 1806 then radiates thetransmission signal as described herein, wherein each unit cell maytransmit individually or transmit as a sub-array.

FIG. 19 illustrates an example of an iMTM antenna array 1900 havingradiating elements 1902, which are each iMTM cells. The array 1900 ofiMTM cells may operate as a single array or may be controlled to operateas multiple subarrays, wherein each of the array or subarrays acts togenerate a radiation beamform that is steerable through control of thereactance of individual iMTM unit cells. The feed structure for the iMTMantenna array structure 1900 is a substrate 1904 having multipleconductive layers and a dielectric layer sandwiched therebetween. Thefeed 1904 is configured as super elements 1906 that are along thex-direction of the iMTM antenna array 1900, wherein each super elementincludes a plurality of slots or discontinuities in the conductive layerproximate the radiating elements 1902. A signal is provided to each ofthe super elements 1906 that radiates through the slots in the superelements and feeds the radiating elements 1902. The various superelements 1906 may be fed with signals of different phase, thus providingphase shifting in the y-direction, while the iMTM antenna array 1900 maybe controlled so as to shift the phase of the transmission signal in they-direction and/or the x-direction, wherein the signal transmits in thez-direction. The ability to control the directivity and phase of thetransmission signal provides flexibility and responsive behavior forwireless communications and radar applications.

In various examples, the iMTM antenna array 1900 may be positionedwithin a vehicle as part of an iMTM radar system (e.g., iMTM radarsystem 100 of FIG. 1), or an infrastructure point within an environment,such as a street lamp or building. In this way, the iMTM array 1900 mayscan the environment with predetermined knowledge of the area, such asroad dimensions, side walk dimensions, traffic signal locations,cross-walk dimensions and so forth. It is appreciated that thedimensions and size provided in the drawings given in these descriptionsis not meant to be limiting, but rather is provided for clarity ofunderstanding of the reader.

FIG. 20 is another perspective of the iMTM antenna array 1900 of FIG. 19illustrating its various layers. Substrate 2000 includes a firstconductive layer 2002, a dielectric layer(s) 2004, and a super elementlayer 2006. The super elements are formed by conductive andnon-conductive traces on a top portion of the super element layer 2006and by vias formed through the super element layer 2006 and through thedielectric layer(s) 2004. The vias (not shown) are lined with conductivematerial, or may be filled with conductive material, so as to formchannels defining the super elements 2012 and providing a wave guidefunction to maintain propagation of the signals fed into the superelements 2012. An optional gap 2008 may be placed between the superelement layer 2006 and the radiating array layer 2010, which containsthe iMTM cells. The longitudinal direction of the super elements 2012 inthe perspective of FIG. 20 is into the page, in the y-direction, withthe signal radiating in the z-direction. Again, note these directionsare for illustration and description purposes only and do notnecessarily correlate to absolute references. Note also that the iMTMarray 2000 may be part of a sensor fusion module (e.g., sensor fusionmodule 1220 of FIG. 12) within the vehicle or infrastructure, wherebydifferent locations share information and communicate with each other toprovide information ahead of action, such as to identify a speeding carseveral blocks before it actually is in range of a given sensor. One ormultiple sensors may provide alerts to other sensors in the environmentto be on the look-out for a speeder.

It is appreciated that the disclosed examples are a dramatic contrast tothe traditional complex systems incorporating multiple antennascontrolled by digital beam forming. The disclosed examples increase thespeed and flexibility of conventional systems, while reducing thefootprint and expanding performance.

The iMTM radar system 100 of FIG. 1 may implement the various aspects,configurations, processes and modules described throughout thisdescription. The iMTM radar system 100 is configured for placement in anautonomous driving system (e.g., autonomous driving system 1200 of FIG.12) or in another structure in an environment (e.g., buildings, billboards along roads, road signs, traffic lights, etc.) to complement andsupplement information of individual vehicles, devices and so forth. TheiMTM radar system scans the environment, and may incorporateinfrastructure information and data, to alert drivers and vehicles as toconditions in their path or surrounding environment. The iMTM radarsystem is also able to identify targets and actions within theenvironment. The various examples described herein support autonomousdriving with improved sensor performance, all-weather/all-conditiondetection, advanced decision-making algorithms and interaction withother sensors through sensor fusion. The iMTM radar system leveragesintelligent metamaterial antenna structures and AI techniques to createa truly intelligent digital eye for autonomous vehicles.

It is appreciated that the previous description of the disclosedexamples is provided to enable any person skilled in the art to make oruse the present disclosure. Various modifications to these examples willbe readily apparent to those skilled in the art, and the genericprinciples defined herein may be applied to other examples withoutdeparting from the spirit or scope of the disclosure. Thus, the presentdisclosure is not intended to be limited to the examples shown hereinbut is to be accorded the widest scope consistent with the principlesand novel features disclosed herein.

What is claimed is:
 1. An Intelligent Metamaterial (“iMTM”) interfacemodule for use with an iMTM antenna module in a radar system,comprising: a target detection module configured to detect a target in aradar point cloud encoded from radar data; a decision neural networkconfigured to determine an action for the iMTM antenna module to performbased on the target detection in the radar point cloud; and a beamcontrol module configured to control the action for the iMTM antennamodule.
 2. The iMTM interface module of claim 1, wherein the targetdetection module comprises a convolutional neural network.
 3. The iMTMinterface module of claim 1, wherein the radar point cloud ispre-processed to correct for non-line-of-sight reflections in anon-line-of-sight correction module.
 4. The iMTM interface module ofclaim 3, wherein the non-line-of-sight correction module comprises aplanar surface identification module and a non-line-of-sight reflectionremapping module configured to generate the radar point cloud for thetarget detection module.
 5. The iMTM interface module of claim 1,wherein the action comprises an instruction to the iMTM antenna modulefor generating a beam with a set of parameters and at a phase shift. 6.The iMTM interface module of claim 5, wherein to control the action forthe iMTM antenna module comprises to determine the set of parameters andthe phase shift for the beam.
 7. An Intelligent Metamaterial (“iMTM”)interface module for a radar system, comprising: an iMTM antenna moduleconfigured to radiate a transmission signal and generate radar datacapturing a surrounding environment; a target detection moduleconfigured to detect and identify a target in a radar point cloudencoded from the radar data; and a beam control module configured tocontrol the iMTM antenna module.
 8. The iMTM interface module of claim7, wherein the iMTM antenna module comprises an iMTM antenna structurehaving a plurality of iMTM antenna arrays.
 9. The iMTM interface moduleof claim 8, wherein the iMTM antenna module is further configured toradiate the transmission signal with the iMTM antenna structure.
 10. TheiMTM interface module of claim 8, wherein each of the plurality of iMTMantenna arrays comprises a plurality of iMTM cells configured into aplurality of subarrays and each of the plurality of iMTM antenna arraysprovides a plurality of phase shifts in the transmission signal.
 11. TheiMTM interface module of claim 8, wherein the iMTM antenna structurecomprises a feed distribution module having an impedance matchingstructure and a reactance control structure.
 12. The iMTM interfacemodule of claim 7, wherein the target detection module comprises aconvolutional neural network that is coupled to a decision neuralnetwork.
 13. The iMTM interface module of claim 7, wherein the radarpoint cloud is pre-processed to correct for non-line-of-sightreflections in a non-line-of-sight correction module.
 14. The iMTMinterface module of claim 7, wherein a data pre-processing module isused for encoding the radar data into the radar point cloud.
 15. TheiMTM interface module of claim 14, wherein the data pre-processingmodule comprises an autoencoder.
 16. A method for using an IntelligentMetamaterial (“iMTM”) interface module for a radar system, comprising:directing an Intelligent Metamaterial (“iMTM”) antenna structure toradiate RF beams with determined parameters; receiving reflections fromthe RF beams; generating, from the reflections, radar data capturing asurrounding environment; identifying a target in a radar point cloudencoded from the radar data; and determining a next action for the iMTMantenna structure.
 17. The method of claim 16, wherein directing theiMTM antenna structure to radiate RF beams with determined parameterscomprises directing a plurality of iMTM antenna arrays in the iMTMantenna structure to radiate RF beams with a set of directions and phaseshifts.
 18. The method of claim 16, further comprising: encoding theradar data into the radar point cloud with non-line-of-sight correction.19. The method of claim 16, further comprising: extracting micro-dopplersignals from the radar data.
 20. The method of claim 19, whereinidentifying the target comprises using the micro-doppler signals and aconvolutional neural network coupled to a decision neural network toidentify the target.